The dataset was retrieved from Kaggle: Hourly energy demand generation and weather. This dataset contains 4 years of electrical consumption, generation, and pricing for Spain.

Description of the Problem

k-Nearest Neighbors (kNN), Linear Regression (including L1 and L2), Decision Tree, Random Forest, and Gradient Boosting Methods (along with its variations), are models that inherently struggle with extrapolation, particularly in the context of time series forecasting. This study aims to develop a specialized Data Preprocessing and Feature Engineering Pipeline tailored for time series challenges, with a focus on forecasting electricity prices. Our objective is to create a process for the models to enhance their ability in interpreting and predicting time-based data through appropriate feature engineering and preprocessing techniques. In this case, it is to be able to predict the hourly prices for the next 21 days.

Highlights

This study delves into solving Time Series Machine Learning issues, focusing on Feature Engineering and Data Preprocessing for such problems. Key aspects include:

1. DateTime Extraction: Extracting and utilizing date-time data.
2. Lag Variable Integration: Incorporating lag variables for better predictions.

The study emphasizes Mean Absolute Error (MAE) as the crucial metric for evaluating Time Series models, given its accuracy in measuring prediction errors.

Methodology: The Machine Learning Pipeline

Data Exploration¶

The data was explored in terms of how the features initially behaved, and how much the independent variables are correlated with each other, and how the independent variables are correlated with the target variable. This section emphasizes the use of the Pearson Correlation Coefficient Heatmap to address potential multicollinearities.

Data Preprocessing: Manual¶

Subsequently executed with the Data Exploration, this section looks into the assesment of NaN Variables and potential Categorical Variables. Unlike the Feature Engineering in the next section, Data Processing was not subjected to automation (i.e., creation of functions).

Feature Engineering: Automated¶

This section tackles the core of the study by integrating variables that are crucial to solving Time Series problems. The column containing information about the date was assessed, then was subjected to extraction by year, month, day, hour, and weekday.

This section also tackles the integration of Lag Variables. Lag Variables enable the model to utilize the previous immediate data points, thereby increasing predictability of the next datapoint.

Subjecting the Final Dataset to the Models¶

Eight models were used for the study, namely:

• k-Nearest Neighbors (kNN)
• Linear Regression
• Linear Regression with L1 Regularization
• Linear Regression with L2 Regularization
• Decision Tree
• Random Forest
• Gradient Boosting Method
• XGBoost

For the purpose of demonstrating the effectiveness of Feature Engineering and meticulous Data Preprocessing, the hyperparameters of the models were left on default.

Execution of Methodology

Data Exploration¶

Data Preprocessing (Manually Done)¶

Addressing Missing Values¶

generation hydro pumped storage aggregated and forecast wind offshore eday ahead are fully NaN values. It should be dropped completely.

Decision: Remove NaN values completely

For this section, only 47 rows were emmitted from the 35,064 rows. 35,017 rows remained.

Addressing Potential Categorical and Ordinal Values¶

From this section, we discovered that there are only zeros in the mentioned columns. They serve no purpose to the model whatsoever. Subsequently, this addressed the initially unusual Pearson Correlation Coefficient Heatmap seen from Data Exploration section. We can compare it to the Heatmap succeeding below this text.

Other Potentially Unnecessary Variables¶

Inspecting the Heatmap with the processed dataset, we see that total load forecast, total load actual, price day ahead are strongly correlated with the price actual, because they refer to future references. Therefore, we should also remove other columns that refer to future information, such as the following: forecast solar day ahead, forecast wind onshore day ahead

The model's purpose would be defeated if we base our forecasts on forecasts.

Feature Engineering (Automated by Function)¶

Extract and Integrate DateTime Variables¶

Integrate Lag Variables¶

The Eight Machine Learning Models¶

Import the libraries¶

Final Pipeline Functions¶

This section only aims to provide a streamlined version of the Pipeline, integrating all the findings and methods done by using the functions found in the Final Pipleine Functions section.

process_df(df) is only applicable to this dataset. Assuming that you add more X variables for the sake of prediction, this is the code you would use before subjecting it to create_lag(df, target_col, lag_hours) and extract_datetime(df, datetime_col)

The add_rolling_window_features(data, target_col, window_size) is placed to ensure integrity of the model and results. However, the specifics of these would not be discussed here.

Creating the Model Assessment: Baseline MAE Function¶

When assessing the model's usefulness, we use Mean Absolute Error (MAE) instead of R-Squared. MAE provides a direct, interpretable metric of average error, making it more suitable for assessing the practical impact of prediction errors, whereas R-squared, being a relative measure of fit, is more about explaining the variance in the dependent variable by the independent variables, and can sometimes be misleading in the context of time series predictive accuracy.

In this case, we use the assumption 'price today, price tomorrow', and this is shown by data[target_col].shift(1)

Apply all the functions to the original dataset¶

This section only aims to demonstrate the streamlining of the Pipeline, in conjunction to the Final Pipleine Functions section.

Assigning X and y¶

Specifically for this part, we have to determine how many days we want to predict ahead. In this case, we want to predict the hourly prices of electricity for 21 days.

AutoML and Results¶

Results and Discussion

In order to assess the model's accuracy and reliability,¶

we have to employe the Mean Absolute Error (MAE), as discussed in the Methodology section. The score represents the average discrepancy of the model. The lower the MAE score, the better. In this case, we have set the baseline to 2.552, and this is obtained by using MAE's function from sklearn.metrics. Below is the equation of the Mean Absolute Error.

$MAE = \sum_{i=1}^{D}|x_i-y_i|$

If the MAE Test score of the model is below the Baseline MAE, then it is considered a reliable and accurate forecasting model.

For the table below, it is ordered from the most accurate to least accurate(lowest MAE to highest), and it is evident that Gradient Boosting Method had scored below the Baseline MAE by about 0.2. This is followed by Random Forest. The rest, while having low MAE scores, are less reliable than the first two. In this case, kNN has also shown its complete inability to extrapolate, unlike the other models.

It is also peculiar to see that lag_1, being the result from the previous hour datapoint, has been the most important feature for the prediction of the succeeding hourly prices. It suggests that short-term historical data plays a crucial role in forecasting future values in all of the tested models. This also indicates that the market or system has a strong temporal component, where recent history is a good indicator of the near future.

Visualising the performance of the Machine Learning models,¶

we compare the predicted values with the actual values. All of the models, except kNN, have almost mirrored the actual results.

Finally, we are interested in how effective the Data Preprocessing and Feature Engineering actually is.¶

Suppose that we subject the raw dataset only to its Preprocessing phase, how well would the ML models do? What if there were no Feature Engineering involved in the first place?

To recap, we used process_df(df), create_lag(df, 'price actual', lag_hours=7), add_rolling_window_features(df, 'price actual', window_size=24), and add_baseline_prediction(df, 'price actual') for the pipeline.

For our analysis, we only subject the dataset to the process_df(df) and immediately fit it to the models.

Looking at the MAE results¶

of the model, and using the same baseline from the original process, we see that all models completely failed to beat the baseline. Furthermore, they had much larger discrepancy, presenting a discrepancy average ranging from PHP 7 to PHP 10 per hour for all the models.

We further observe how evident the models fail to predict without the employment of Feature Engineering,¶

by visualizing again the actual target values versus the predicted values.

By implenting the models without any Feature Engineering to the data, we have proved the extreme significance of implementing these first to the data before subjecting them to the Machine Learning models.¶

Conclusion

From the discussions, we have effectively presented how Data Preprocessing and Feature Engineering are crucial to solving Time Series problems. A comparative analysis highlighting the presence versus absence of Time Series Feature Engineering clearly demonstrates its positive impact on model performance. This is particularly notable as the improvements were achieved even without customizing the parameters of the eight evaluated models, implying the inherent efficacy of these preprocessing techniques.

Recommendation

For future applications, two pivotal factors must be given priority: ensuring data stationarity and the practical applicability of the methodology in real-world scenarios.

1. Stationarity: Models often struggle with non-stationary datasets. To address this, applying Time Series Feature Engineering methods like 'differencing' is recommended to stabilize the dataset.

2. Real-World Applicability: In real-world scenarios, some input variables may be projections rather than observed values. This necessitates a rigorous model evaluation, possibly through Cross-Validation techniques tailored for Time Series data, to ensure robustness and reliability.

What to explore next?

The next phase of exploration should concentrate on models specifically designed for Time Series analysis. These models should inherently possess mechanisms to utilize outputs from previous time steps as inputs for future predictions. Such a capability, essentially creating automatic lag features, will significantly enhance the model's proficiency in handling Time Series data, making it more adept at forecasting with temporal dynamics in mind.

References

Brownlee, J. (n.d.). Feature selection for time series forecasting with Python. Machine Learning Mastery. Retrieved from https://machinelearningmastery.com/feature-selection-time-series-forecasting-python/

DotData. (n.d.). Practical guide for feature engineering of time series data. Retrieved from https://dotdata.com/blog/practical-guide-for-feature-engineering-of-time-series-data/

Jhana, N. (n.d.). Energy Consumption, Generation, Prices, and Weather Dataset. Kaggle. Retrieved from https://www.kaggle.com/datasets/nicholasjhana/energy-consumption-generation-prices-and-weather?select=energy_dataset.csv